Lost your marbles? The puzzle of dependent measures in experimental pragmatics

A rarely discussed but important issue in research on pragmatic inference is the choice of dependent measure for estimating the robustness of pragmatic inferences and their sensitivity to contextual manipulations. Here we present the results from three studies exploring the effect of contextual manipulations on scalar implicature. In all three studies we manipulate the salient question under discussion and the perceptual availability of relevant set sizes. The studies differ only in the dependent measure used: Exp. 1 uses truth judgements, Exp. 2 uses word probability ratings, and Exp. 3 uses a direct measure of sentence interpretation. We argue that the first two are effectively measures of production, and find they are sensitive to our contextual manipulations. In contrast the interpretation measure shows no effect of context. We argue that this methodologically troubling finding can be understood and predicted by using the framework of probabilistic pragmatics.